Method and apparatus for real-time viewer interaction with a media presentation

ABSTRACT

Real-time participation within a media presentation is disclosed. A participant overlay may be generated by preprocessing a media presentation to extract a time-variant metadata stream associated with one or more interactive regions depicted in the media presentation. The participant overlay includes simulated physics, which may include boundaries of the interactive regions defined in accordance with content in said media presentation. A participant&#39;s interaction with said media presentation through the interactive regions may be captured and combined with the media presentation for display on a display. The interaction may comprise a user action upon one or more elements in the media presentation, a user action expressed in the media presentation through an avatar, or a user action in said media presentation in connection with a game element.

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application Ser. No.12/419,248, filed Apr. 6, 2009, the entire contents of which areincorporated herein by reference. U.S. patent application Ser. No.12/419,248 claims the priority benefit of U.S. Provisional PatentApplication No. 61/047,737, to Gary M. Zalewski, filed Apr. 24, 2008 andentitled System and “Method For Converting a TV Show Or Movie into anInteractive Game Experience”, the entire contents of which areincorporated herein by reference. This application likewise claims thepriority benefit of U.S. Provisional Application No. 61/047,737.

FIELD OF THE INVENTION

Embodiments of the invention relate to collaborative viewing of a mediapresentation. More particularly, embodiments of the invention relate toconverting a media presentation, such as television show or movie intoan interactive game experience.

BACKGROUND OF THE INVENTION

An annoying aspect of seeing a movie at a theater is distraction causedby conversations held between viewers during the presentation of themovie. It is not that conversation in and of itself is annoying so muchas that it is someone else's conversation. On the other hand, it isenjoyable to share remarks, observations, and jokes with friends andfamily while watching a movie at home. To this end, a popular televisionseries, Mystery Science Theater 3000, was based upon conversations thatwere held between viewers of a movie, typically an appallingly badmovie. In Mystery Science Theater 3000, two robots and a human spacepilot commented and made jokes while the movie was presented. The moviewas shown on a movie screen, as though in a movie theater, and therobots and human commentators appeared in silhouette, as though theywere seated in the movie theater watching the movie.

It is known to provide players with the ability to talk to one anotherwhile playing a video game over a network. In fact, headsets areavailable for this purpose. As is known, players have a persona oravatar in such games that is a representation of the player. Thus, aplayer may participate in a game and talk with other players while doingso.

In the case of one or more individuals passively viewing a mediapresentation, such as a movie, there is no way to interact with themovie within the movie space itself. That is, those individuals viewingthe movie cannot be a part of the presentation of the movie outside ofoffering up comments in their living room or in the movie theater.Simply put, they are not a part of the action on the screen. This isdue, in part, to the fact that a media presentation, such as a movie, isstatic in that it exists with a beginning, middle, and end before it ispresented to an audience.

In the case of a video game, players are participants in the unfoldingdrama of the game and cannot act as bystanders. This is due, in part, tothe fact that a video game is dynamic in that the narrative of a videogame is created in real-time as the players interact with the game.

B. Burrill, T. Kirste, J. Weiss, Time-varying Sensitive Regions inDynamic Muti-Media Objects: a pragmatic approach to content-basedretrieval from video, Information and Software Technology, Vol. 36, No.36, pp. 213-223 (1994) describe a general concept for accessing thecontents of the dynamic media by two points of view, i.e. those ofhyper-media and information retrieval. Burrill et al. introduce thegeneral concept of Sensitive Regions (or “hot-spots”) by reverseengineering techniques from established technologies. Burrill et al.also describe three applications being developed which explore a varietyof aspects associated with Sensitive Regions, i.e. theHyperPicture-System which manages such data, MOVie experiments with thecreation of editing of Sensitive Regions in the cinematically orientedcontext, and Share ME which explores issues associated with the use ofSensitive Regions in the interface to multi-media applications.

D. Wistendahl, L. Chon, System for mapping hot-spots in media contentfor interactive digital media programs, U.S. Pat. No. 5,708,845 (13 Jan.1998) disclose a system for allowing media content to be used in aninteractive digital media program. Such system comprises Frame Data fromthe media content in object mapping data (N Data) representing the frameaddresses and display location coordinates for objects appearing in themedia content. The N Data are maintained separately from the Frame Datafor the media content, such that the media content is kept in tactwithout embedded codes and can be played back on any system. The systemestablishes linkages connecting the object mapped by the N Data theother functions to be performed in conjunction with display of the mediacontent. Selection of an object appearing in the media content withpointer results in initiation of the interactive function. Accordingly,a broad base of existing non-interactive media content, such as movies,videos, advertising, and television programming can be converted tointeractive digital media use.

While the art provides a rudimentary scheme for interaction with mediathrough various mapping techniques, there has been no attempt in the artto address the effects of such interaction, such that the interaction iscoordinated with the content in a realistic way.

It would be advantageous to provide an experience in which one or moreindividuals could interact with a media presentation without previouslybeing a part of the presentation itself.

SUMMARY OF THE INVENTION

The disadvantages associated with the prior art are overcome byembodiments of the present invention which are directed to a method andapparatus that allows real-time user interaction with a mediapresentation.

In an embodiment of the invention, users are enabled to interact withenhanced interactive productions of television shows or movies, basedupon adding game overlays to such movies or television shows. In thisway, the physics in the game or virtual world meet with the boundariesdefined in the television show or movie set. For example, avatarsdisplayed to a user, in response to user gestures in the real world,e.g. in response to manipulation of a game controller or other suchexpedient, may toss tomatoes that stick to the actor's face or bounceoff the actor's face and roll along a table that appears in the movie ortelevision show.

In an embodiment, the movie or television show may be preprocessed toextract a time-variant metadata stream associated with regions in themoving pictures. An overlay having both active regions and addedcontent, such as animation, is composited with the movie or televisionshow. Such preprocessing can include both visual and/or aural componentsof the movie or television show.

User interaction with the enhanced movie or television show via a gamecontroller or the like may be mapped to the movie or television show viathe overlay, and game or other active features of the overlay respondduring presentation of the movie or television show.

As a result, the audience can be brought into, and interact with, theperformance depicted in the movie or television show. Such interactionmay include, for example, individual user actions, such as throwing atomato, as discussed above, or group and/or competitive actions, such asshooting a spider that is crawling on an actor's back.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIGS. 1A-1C provide a series of block, schematic diagrams that showprophetic examples of user interaction with a media presentation throughmanipulation of an object within the presentation space according to anembodiment of the invention;

FIG. 2 is a block, schematic diagram illustrating a prophetic example ofa system for real-time viewer interaction with a media presentationaccording to the invention;

FIGS. 3A-3C provide a series of block, schematic diagrams that show aprophetic example of user interaction with a media presentation throughaction of a user avatar within the presentation space according toanother embodiment of the invention; and

FIGS. 4A-4C provide a series of block, schematic diagrams that showprophetic examples of user interaction with a media presentation as partof a game that is appended to the media presentation within thepresentation space according to the invention.

FIG. 5 is a flow diagram depicting a prophetic example of a method forreal-time participation within a media presentation in accordance withan embodiment of the present invention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the exemplary embodiments of the invention described below are set forthwithout any loss of generality to, and without imposing limitationsupon, the claimed invention.

The disadvantages associated with the prior art are overcome byembodiments of the present invention which are directed to a method andapparatus that allows real-time user interaction with a mediapresentation.

In an embodiment of the invention, users are enabled to interact withenhanced interactive productions of media content, such as televisionshows or movies, based upon adding game overlays to such movies ortelevision shows. In this way, the physics in the game or virtual worldmay include boundaries defined in terms of content elements of thetelevision show or movie. For example, avatars displayed to a user, inresponse to user gestures in the real world, e.g. in response tomanipulation of a game controller or other such expedient, may tosstomatoes that stick to the actor's face or bounce off the actor's faceand roll along a table that appears in the movie or television show.

In an embodiment, the media content, e.g., a pre-existing movie ortelevision show, may be preprocessed to associate a time-variantmetadata stream with regions in the moving pictures. An overlay havingboth active regions and added content, such as animation, is compositedwith the movie or television show. Such preprocessing can include bothvisual and/or aural components of the movie or television show.

By way of example, the time variant metadata stream may be associatedwith the media content in the manner described in U.S. Pat. No.5,708,845, which is incorporated herein by reference in its entirety.Specifically, frame data for a digitized version of the media, e.g.,video, content, may be associated with object mapping data (N data)representing frame addresses and display location coordinates of objectsappearing in the media content. An interactive digital media (IDM)program may be used to establish linkages connecting objects mapped bythe N data to interactive functions to be performed in conjunction withdisplay of the media content.

User interaction with the enhanced media content via a game controlleror the like is mapped to the media content via the metadata overlay, andgame or other active features of the overlay respond during presentationof the movie or television show.

As a result, the audience may be brought into, and interact with, theperformance depicted in the media content. Such interaction may include,but is not limited to, for example, individual user actions, such asthrowing a tomato, as discussed above, or group and/or competitiveactions, such as shooting a spider that is crawling on an actor's back.

FIG. 5 shows a flow diagram generally illustrating a prophetic exampleof a method 100 for real-time participation within a media presentation.The method 100 may be implemented in the form of a set of computerprocessor executable instructions, which may be embodied in a computerreadable storage medium such as a hard disk, flash memory, CD-ROM, orrandom access memory (RAM). A participant overlay is generated asindicated at 102. By way of example the participant overlay may begenerated by preprocessing a media presentation to extract atime-variant metadata stream associated with one or more interactiveregions depicted in the media presentation. The participant overlay mayinclude simulated physics. The simulated physics may include boundariesof the one or more interactive regions defined in accordance withcontent in the media presentation.

A participant's interaction with the media presentation through the oneor more active regions is captured as indicate at 104. The interactionmay comprise, e.g., a user action upon one or more content elements insaid media presentation, a user action expressed in said mediapresentation through an avatar, or a user action in the mediapresentation in connection with a virtual object, such as a gameelement. The participant's interaction may be captured by way of acontroller, such as a game controller, that is used in conjunction witha system used to display the media presentation and participant overlay.The overlay (including participant interactions) is combined with themedia presentation for display to a participant on a display asindicated at 106. The media presentation and overlay may then bedisplayed simultaneously on the display as indicated at 108.

FIGS. 1A-1C provide a series of block, schematic diagrams that showing aprophetic example of user interaction with a media presentation throughmanipulation of an element within the presentation space according to anembodiment of the invention. FIGS. 1A-1C have in common an audience 10a,10 b, which in this example comprises two audience members, each at acommon location, and each having the ability to interact with apresentation of a movie or television show via a game controller. Thoseskilled in the art will appreciate that any number of individuals at anynumber of locations may comprise an audience, e.g. anywhere one personat one location to many individuals at many locations. Further, userinteraction may take place with any or game controllers, motion sensingdevices, and the like, and combinations thereof.

Further in common in FIGS. 1A-1C are a display 11 in which the audienceis represented as avatars 15 a, 15 b; a score panel 13 which isdisplayed at least periodically in those embodiments of the inventionthat involve competition; and an object tracking module 12. In thisembodiment, the object tracking module 12 provides data to a motionsensor 14 and thus enables actions of the audience in the real world tobe represented in a temporal and physically correct manner in thepresentation. By way of example, and not by way of limitation, theobject tracking module 12 may additionally implement tracking ofelements (e.g., objects or characters) within the media presentation.For example, the object tracking module may implement face tracking ofthe faces of one or more characters appearing in the media presentationto generate metadata (e.g., N data) associated with the tracked faces.

In an embodiment, the movie or television show may be preprocessed toextract a time-variant metadata stream associated with regions in themoving pictures. By way of example, a generator of the metadata stream,referred to herein as an author, may identify one or more elements in aparticular frame of digital media content (or other media content thathas been digitized) using an outlining tool in a video editing softwaresuch as ELASTIC REALITY 3™ from Avid Technology, Inc. of Tewksbury,Mass. If the element remains stationary throughout a number ofsucceeding frames, the author need not identify the element separatelyfor each frame. Alternatively, if the element moves relative to theframe, a motion tracking or machine vision tool may be used toautomatically identify the object in successive frames. An example ofsuch a motion tracking tool is known as ASHET-2 developed by Stephen M.Smith at the U.K. Defense Research Agency, Chertsey, Surrey, UK. Thisprogram uses feature segmentation and clustering techniques to producean abstracted representation of elements in successive frames of a videosequence. Using statistical comparisons, a cluster characterized by asimilar set of features appearing at different positions in a pathacross a series of frames can be recognized as an element in motion.Once an element has been identified, a hyperlinking tool may be used toestablish hyperlinks to the metadata to allow interaction with theelements.

To facilitate interaction between the audience members 15 a, 15 b andthe elements in the media content, the display 11 may operate inconjunction with an interactive device, such as a computer or video gameconsole device. Each audience member 15 a, 15 b may operate aconventional game controller to provide control signals to theinteractive device. The interactive device may be configured to permitinteraction with interactive objects presented on the display 11, e.g.,through application-specific hardware or through software programming.The interactive objects may include elements (e.g., objects orcharacters) appearing in the media content that have associatedmetadata. Such elements are referred to herein as “content” elements. Inaddition, the interactive objects may include objects that are generatedby the interactive device for the specific purpose of interacting withthe content elements. Such objects are referred to herein as “virtual”objects.

In embodiments of the present invention, the metadata and/or interactivedevice may be configured to define one or more parameters of simulatedphysics that govern the interaction between virtual objects and contentelements defined by the metadata. By way of example, and not by way oflimitation, such physics parameters may include physical boundaries thatdefine one or more regions of a scene presented on the display 11 thatare subject to interaction. These physics boundaries may includevertical boundary surfaces, such as walls, or horizontal boundarysurfaces, such as floors, ceilings, tabletops, and the like. Thesephysics boundaries may be associated with corresponding content elementsdisplayed in a scene shown on the display 11. By way of example and notby way of limitation, the physics boundaries may conform to the outlineof a character depicted in the media presentation. The physicsparameters may define the nature of interactions between virtual objectsand the boundaries, e.g., the degree to which collisions between themare elastic or inelastic.

The physics parameters may also define relevant physical quantitiesgoverning the virtual objects. Specifically, the physics parameters maydetermine a direction and magnitude of the force of a gravity vectorgoverning motion of virtual objects. The gravity vector may be orientedin any suitable fashion, e.g., parallel to the vertical edges of theframe or aligned with respect to the scene displayed in the frame.Furthermore, the physics parameters may define optical parameters ofvirtual objects, such as transparency, reflectivity, index of refractionand the like. In the case of solid virtual objects the physicsparameters may include elasticity, plasticity, yield points, breakingstrengths, and the like. In the case of virtual objects that simulateliquids the physics parameters may include density, viscosity, surfacetension and the like.

The physics parameters may also define when and how a user may interactwith a content element or interactive region shown on the display. Forexample, the physics parameters may define one or more regions of ascene or frame, referred to herein as “blackout” regions whereinteraction is prohibited. The parameters may also define one or moreregions, referred to herein as “target” regions where interaction isexpressly invited. Furthermore, the physics parameters may defineregions of a scene, referred to herein as “free” regions, where virtualobjects may appear and move.

As an example of interaction between a content element and a virtualobject, audience member 10 b may have a corresponding representationdisplayed on the display along with the media presentation in the formof an avatar 15 b. As the audience member 10 b operates his gamecontroller to direct the avatar to lift its arm in the presentation, theavatar 15 b can responds accordingly. In FIG. 1A, the audience member 10b operates a game controller to begin to throw a virtual tomato 16 at anactor in a movie or television show. The actor's face may be a contentelement having an associated metadata stream that permits interactionwith the virtual tomato. As depicted in FIG. 1B, the audience member 10b may fling the virtual tomato 16 at the face of the actor and thetomato may be depicted in the presentation, as well as the throwinggesture of the avatar 15 b for the audience member 10 b.

As shown in FIG. 1C, the tomato may land on the actor's face and is drip17 onto the actor's shirt and jacket. Metadata associated with thevirtual tomato 16 and/or the actor may govern the physics of thesmearing and dripping of the tomato. By way of example, if the throw bythe audience member 10 b is accurate, his score may be incremented.Thus, the score for Gary may be incremented to 111.

This embodiment, as shown for purposes of example in FIGS. 1A-1C,provides for user interaction with a media presentation throughmanipulation of an object within the presentation space. Any object canbe manipulated relative to any portion of a presentation. The example ofa user throwing a tomato at an actor's face is only one possibility ofan almost infinite number of options. For example, a user may pick up achair and break a window; a user may drive a car over a cliff, etc.

In some embodiments, the nature of the interactivity may be based on thenature of a content element of the media presentation. For example, morepoints may be awarded if a virtual tomato hits a character who is the“star” of the media presentation.

Furthermore, virtual objects may serve a purpose that is ancillary totheir interaction with the media presentation. For example, a virtualobject may incorporate advertising information, which can be displayedon the display. Such virtual objects are sometimes referred to herein as“brand name” objects

FIG. 2 is a block schematic diagram of a system for real-time viewerinteraction with a media presentation according to an embodiment of theinvention. FIG. 2 provides an example of an embodiment in which eachparticipant is interconnected via a network 24, such as the Internet.Other embodiments do not need a network connection and may be entirelylocal, as will be discussed below. In FIG. 2, several participantlocations 20 and 20 a-20 n are shown, which provide facilities for theirrespective users 22 and 21 a-21 n. By way of example, and not by way oflimitation, participant location 20 may include a television or othertype of display 33. The participant at this location 20 may interactwith the system, e.g., by means of a game controller 28, microphone 29,and video camera 30. Embodiments of the invention contemplate otherforms of user interaction as well, such as a cell phone, where theinteraction takes place, for example, amongst theater goers in a movietheater. The interaction of the participant is captured at a localprocessor 27, which may be implemented by a general-purpose computer orother device having a computer processor. By way of example, the localprocessor 27 may be implemented in a game console, such as a SonyPlayStation 3.

Each participant location may be interconnected via the network 24 witha game server 32 and a compositing server 31. The game server 32 may beprovided in those embodiments that provide game play features, asdiscussed below. The compositing server 31 provides an overlay OL onto amedia presentation MP that allows interactive features to be added tosuch media presentations. In some implementations, the functions of boththe game server 32 and the compositing server 31 may be combined in asingle server device. Further, the game server and/or compositing serverfunctions can be implemented at a participant's location, for example bythe local processor 27. In this embodiment, a network is not necessary.However, the inclusion of a network provides interaction from aplurality of participants at a plurality of remote locationssimultaneously and in real time.

A media server 26 may provide content 25 corresponding to the mediapresentation MP to the compositing server 31. Alternatively, the content25 may be provided through other means. For example, the content 25 maybe embodied in a computer readable medium such as a digital video disk(DVD), CD-ROM, universal serial bus (USB) flash drive and the like andaccessed by the local processor 27 through an appropriate interface,e.g., a DVD player, CD-ROM drive or USB port. The compositing server 31combines the participant interactions with the media presentationcorresponding to the content 25 in real time. Time codes and other knownsynchronization techniques may be used to assure accurate registrationof user interactions with the media presentation, and may also be usedto map the media presentation, for example on a frame-by-frame or scenebasis, to add elements to the media presentation and to track userinteractions on the basis of locations within the media presentation.For example, a mapping may be used to track user movement of objects,shooting, avatars, and the like in the overlay in accurate registrationwith the underlying media presentation MP. Such techniques as collisiondetection and the like, as are well known to game developers of ordinaryskill in the art, may be applied to implement these features.

As shown in FIG. 2, the media presentation 25 may appear on aparticipant's display 33 as a video presentation. Likewise, avatars forthe various participants 21 and 22 are also shown on the screen asavatars 21 a and 22 a. Actions of the participants are shown in asimilar way. Also note that FIG. 2 shows a split screen, in which themedia presentation appears in an upper portion of the screen and theparticipants appear in a lower portion of the screen. This convention isnot required, and the overlay may be generated as desired, e.g. over theentire screen.

Accordingly, an embodiment of the invention overlays an interactiveelement onto a media presentation in real time, i.e. while it isplaying. In one embodiment, a scene or other narrative element ispresented as part of the media presentation. The scene or narrativeelement may be a scene from existing content, such as a movie,television show, music video, sporting event, or the like; it may be aprepared piece of content for use in connection with the interactivesystem described herein; or it can be a video game or other interactiveactivity, e.g. where spectators to a video game may interact with thegame play as an audience. While the foregoing example mentions that ascene is presented, embodiments of the invention contemplate that anentire movie, for example, may be presented. By way of example, and notby way of limitation, the media presentation may comprise a standardcable or satellite broadcast, over-the-air digital or analog terrestrialbroadcast, streamed media, DVD, or Blu-ray presentation as a contentsource.

FIGS. 3A-3C provide a series of block, schematic diagrams that show userinteraction with a media presentation through action of a user avatarwithin the presentation space according to another embodiment of theinvention. In the example of FIGS. 3A-3C, a scene presented on a display33 shows an actor 37 walking along a sidewalk. An interactive layer iscomposited with the scene to superimpose a target 38 on the actor'sbehind that is visible to all individuals who are participating in aninteractive, real time viewing session. In one example, the display 33shows the content scene and the target, as well as all members in anaudience, for example, in a theater watching the scene, including thosemembers of the audience 35 a, 35 b who are interacting with thepresentation. In this example, one member of the audience 31 ispassively viewing the presentation. The target provides a game elementand cues the audience to take some action. For example, each interactingmember of the audience could throw a virtual spitball at the screen thatother participants could see (see the example described above inconnection with FIGS. 1A-1C).

Each interacting member of the audience 35 a, 35 b may have acorresponding avatar 39 a, 39 b shown on the display 33 along with thepresentation. As depicted in FIG. 3A, each interacting member of theaudience may be shown with their corresponding avatar seated. As seen inFIG. 3B, an audience member 35 a may use a game controller or other suchmechanism to move his avatar 39 a out of its seat in the presentationand to have the avatar walk up to the target. In the prophetic exampleshown in FIG. 3C, the avatar of the audience member 39 a is now out ofits seat, has rushed up to the actor in the scene, and has kicked theactor in the behind while the target is still displayed. The firstparticipant to accomplish this may be awarded points. Total points aredisplayed in a ranking among the players 40. In this prophetic example,Gary has successfully completed this action and his score is incrementedto 111. Alternatively, a player may be given some other acknowledgement,or could act solely for his, and everyone else's, entertainment.

Key to this embodiment of the invention is the adding of arepresentation of an audience member, e.g., in the form of an avatar, tothe presentation. Enhancements to movies or television shows may be bothoverlaid and tied temporally and physically to the action within thepresentation. In the example depicted in FIGS. 3A-3B, the target tracksthe movement of the actor. Accordingly, in this example, the targetappears to be static with respect to the actor's frame of reference; andthe avatar moves within the presentation space in a realistic way, suchthat the avatar can be manipulated to match the actor's stride, or torun up to the actor as the actor is moving and, for example, kick thetarget.

In one embodiment, a system is provided that enables actions inconnection with specific interactive objects at referential locations ona display, for example on a television or computer display. Thelocations for the interactive objects and the times at which they arepresented during a particular media presentation may be predetermined sothat presentation of the objects makes sense for that particular mediapresentation. For example, objects may be presented at any time duringwhich user interaction would enhance the entertainment value of themedia presentation, such as during a chase scene, during a love scene,during a fight, or as voted upon by the audience, for example, during aboring part of the media presentation, when the audience is restless. Byway of example, and not by way of limitation, participants may signaltheir inattention by posting an icon resembling a popcorn box on thedisplay. This is similar to going to the lobby in a movie theater. If apredetermined number of popcorn boxes are displayed, then interactivefeatures are turned on.

Alternatively, the media presentation may be automatically enabled foroverlay viewer interaction; or a palette of actions can be persistentlydisplayed or user invoked, by which any viewer can add interactiveelements to a media presentation. In this later case, the user maycomment with a voice bubble, take an action, such as throwing an object,move and animate an avatar, display a reaction, such as approval ordislike, and so on.

As discussed above, boundaries can be applied that restrict the area ofuser interaction, for example to an audience. Further, each member ofthe audience may have a personal element, such as an avatar, so thatthey are recognized by other members of the audience. In anotherembodiment, metadata is provided that defines referential corners of anobject. For example, the metadata may identify areas within thepresentation, such as a person's behind, upon which a target wouldappear (see FIG. 3).

An embodiment of the invention allows, for example, a movie distributorto create a game for the movie. At some point in time during the movie,the game element is aware, for example, that there is a table at aparticular location in the frame, or that a person is bending over.Metadata tells the system the boundary points of this element. Eachparticipant that is receiving the movie thus knows which screen elementsare active based upon the coordinates of the object in the frame. Userinteraction with the object may be tracked locally, for example, using avideo game connected to the user's television and, optionally, to anetwork. The participant's interaction may be composited with the mediapresentation in real time and, based upon the coordinates of the objectin the frame, the participant's actions may be merged into the frameand, alternatively, used to generate a score if the participantsuccessfully performed a task, such as hitting a target or reactingbefore any other participant. This later case might be applied, forexample, in a game such as Where's Waldo, in which the first participantto react to the appearance of a character or object would receivepoints.

Further, user interaction may be captured and used to prepare ananalysis of the media presentation. For example, those scenes that weremost engaging to the audience could be identified.

FIGS. 4A-4C provide a series of block, schematic diagrams that show userinteraction with a media presentation as part of a game that is appendedto the media presentation within the presentation space according to theinvention. One example of this embodiment is a game implementation thatprovides a shooting game. For example, a metadata stream may add avirtual spider 48 to the narrative of a science-fiction or horror movie.The virtual spider may jump on an actor in the scene from time to time.When the virtual spider appears, it may begin crawling up the back ofthe actor on which it has landed. The metadata stream may be configuredto realistically match motion of the spider to motion of the actor.Boundaries may be established that the spider cannot go beyond due tothe presence of another object in the scene, such as a person, a book onthe table, a glass, etc. The metadata stream may define referenceswithin the frame, which establish constraints within the time base ofsequenced images where the game elements are overlaid and mixed in withthe media presentation. In this example, participants can shoot at thevirtual spider.

In some embodiments, the metadata stream may include an audio transcriptthat is time referenced to action taking place in the media presentationso that interactions with virtual objects may be triggered by eventsoccurring in the media presentation. For example, the virtual spider maybe triggered to jump if a character in the media presentation says“jump”.

In the prophetic example shown in FIG. 4A, one audience member 35 a hasa game controller or other such device that is used to shoot at thespider. Other audience members 41 may passively view the presentation.As an actor 37 walks through a scene in the media presentation, thevirtual spider 48 moves with the actor, and also moves up the actor'sback to bite the actor in the neck. Progress of the spider may proceedas shown in FIG. 4B. In the prophetic example depicted in FIG. 4C, theaudience member 35 a has shot at the spider and hit it. This isindicated in this embodiment by the spider exploding 49. Because thespider was hit (and the actor spared the fate of having his neckbitten), the score 40 of the audience member is incremented. Thus,Gary's score is incremented to 111.

The media presentation may be viewed simultaneously by many people atmany locations. At the same time, the participants are also playing agame. Each participant may be connected to the overlay generator by anetwork, such as the Internet. In another embodiment, an overlay ofaliens may be displayed, the overlay generator may establish conditionsand an environment upon which a game/film producer overlays the specificmini-interactive event, in this case the aliens may jump up and down ona table. As the participants attempt to shoot the aliens, the gameserver may track user actions and records participant scores, which aredisplayed both to each participant on his television, and which can alsobe displayed as a ranked list of the highest scoring participants.Animation is added to the overlay to show the effect of shooting eachalien.

According to another embodiment an interactive piece may be added to amedia presentation by repurposing existing content. For example, aninteractive layer may be added to an existing movie, such as a warmovie. The interactive layer may designate characters in the movie thatare shootable, or shootable (virtual) characters may be added to themovie, as in the alien example above.

An embodiment provides an interactive participant score register inwhich participant interactions are kept. A participant's score and/orprevious interaction may affect a subsequent segment of play. Forexample, if a participant killed an alien successfully, then the nextalien might be a bigger monster that is harder to kill.

Another embodiment may provide an outline of the participants' avatarsin a theater overlay, as discussed above. However, in this embodiment,the interactive aspect may be sponsored. For example, the overlaygenerator composites images of a sponsoring company's products, such asreal Coca-Cola cans. The participants could take these cans and throwthem at targets during interactive portions of the media presentation.The participant interaction and gestures may be based upon a suitableform of input. Examples of such input include, but are not limited tooperation of a game controller, participant motions and gesturescaptured by a camera that is observing the participant, or sounds madeby a participant and captured by a microphone. In this embodiment, theoverlay may comprise a frame on the bottom of the screen with theaudience depicted in the frame. Each participant may be represented byhis avatar as a member of the audience depicted in the frame.

Participants who wish to interact with the media presentation may use agame controller or make a recognized gesture, e.g., trigger the throwingof a virtual object at a portion of media presentation. Theparticipant's avatar may be depicted on the screen as throwing thevirtual object into the media presentation depicted on the screen. Inthis prophetic example, since every participant's avatar is depicted inthe frame, each participant can see every other participant'sinteraction with the scene, e.g., people throwing objects on the screenand trying to hit the target while watching. By way of example,participants may throw virtual tomatoes because they really do not likea certain scene. In this prophetic example, the number of tomatoes maybe counted for each scene and this information may be provided as areport to a producer of the show, indicating that a particular scene wasnot popular, was popular, was engaging, etc. Thus, this embodiment maybe useful for capturing demographic information with regard to a viewingaudience.

As discussed above, a key aspect of certain embodiments of the inventionis frame level physics-based interaction of a user with one or moreobjects in the media presentation. That is, users may be presented anoverlay, such as a game overlay, and may interact with an underlyingmedia presentation in a way that conforms to the laws of physics (orphysics simulation). Thus, as in a previous example, a virtual tomatolanding on the face of person depicted in the media presentation maycreate an impact in which the virtual tomato naturally deforms andsplatters. Furthermore, the virtual tomato may then drip from thedepicted person's face in a manner that is natural and realistic.

A further embodiment of the invention contemplates interaction of anelement of a media presentation with a game element. In the propheticexample of the virtual tomato, the tomato hits the depicted person'sface, but the depicted person's face is not necessarily deflected by thevirtual tomato. In this further embodiment of the invention, theindividual's face may be deflected.

To facilitate such interaction, embodiments of the invention may includea mechanism for extracting an element of an underlying mediapresentation and remapping an animated version of that element into thescene. In this regard, consider a prophetic example where the underlyingmedia presentation is a vintage 1920's Charlie Chaplin film in whichCharlie Chaplin is drinking from a clear glass that might contain water,all while he is walking around a set during the scene. In thisembodiment, the individual is provided would be provided with theability to use a joystick or other game controller to throw animatedrubber ducks into the glass. Not only would the individual score points,should the duck land in glass, but the water in the glass would rise involume as the ducks landed in the glass. Further, water might splashfrom the glass, and a duck landing on the edge of the glass, but stillwithin the glass, might bounce from the side of glass into the water,displacing water appropriately. To accomplish this, the water originallydepicted in the glass in the Charlie Chaplin film may be rendered, forexample using wire frame or other animation techniques. The renderedobject replaces the original object in the presentation, for example aspart of the overlay. The overlay may track the original object as theobject moves from frame-to-frame in the underlying media presentation.Thus, Charlie Chaplin would be walking around with a glass of water inwhich the water appears exactly as in the original presentation becauseit is a rendering of that water based upon the underlying image, but inwhich it has been modified to exhibit animated effects. Thus, an objectmay be introduced into a media presentation as part of a game overlayand that object may interact with another object that is part of theunderlying media presentation as well. The object that is originallypart of the media presentation, having been re-rendered in aninteractive form, would not appear any differently than in the originalpresentation until such time as there is a user interaction with theobject. Thus, as mentioned above in connection with the glass of water,a rubber duck landing into the water would both raise the level of thewater in the glass and also cause water to splash from the glass.

Those skilled in the art will appreciate that this technique may be usedin many different ways. For example, as mentioned above in connectionwith the example of a tomato being thrown at an individual, the blow ofthe tomato glancing off the individual's face may cause the individual'sface to be displaced appropriately. In the example of an individualkicking a character in a presentation, the kicked individual may bedisplaced or the portion of the body that is kicked may respond. Key tothis aspect of the invention is modeling the physics and the interactionof the impacting object, e.g. the duck and the underlying object, e.g.the water, in a way that is realistic. Known game technology techniquescan be used to accomplish this modeling.

A key aspect to certain embodiments of the invention is the compositingof an interactive game layer over an existing piece of media content,and in this aspect of the invention, the replacement of some element ofthe media presentation with a version thereof that is capable ofinteracting in an animated and realistic fashion, based upon the physicsof the interaction. In one embodiment of the invention an editor mayrepurpose selected portions of the underlying content to add metadata atthe frame level that re-renders those portions which are intended to beactive on the basis of the laws of physics. In a broader sense, thisaspect of the invention applies metadata to a content presentation todefine boundaries of objects in the video stream based on physics of theobjects themselves. Thus, in another embodiment, advertising assets maybe added to either overlaid objects as part of the game overlay, or as areplacement for objects in the underlying content presentation. Forexample, a box of cereal, which appears in a scene where a family ishaving breakfast, could be replaced with a box of cereal that isphysically identical, but that includes an advertising logo of a cerealmanufacturer. Similarly, a logo, such as the Nike logo, could be addedto a pair of shoes. The invention would track movement of the individualas the individual walked or ran through the scene and apply the Nikelogo in a realistic way.

While the above is a complete description of the preferred embodiment ofthe present invention, it is possible to use various alternatives,modifications and equivalents. Therefore, the scope of the presentinvention should be determined not with reference to the abovedescription but should, instead, be determined with reference to theappended claims, along with their full scope of equivalents. Any featuredescribed herein, whether preferred or not, may be combined with anyother feature described herein, whether preferred or not. In the claimsthat follow, the indefinite article “A”, or “An” refers to a quantity ofone or more of the item following the article, except where expresslystated otherwise. The appended claims are not to be interpreted asincluding means-plus-function limitations, unless such a limitation isexplicitly recited in a given claim using the phrase “means for.”

Throughout this description, the embodiments and examples shown shouldbe considered as exemplars, rather than limitations on the apparatus andmethods disclosed or claimed. Although many of the examples presentedherein involve specific combinations of acts or system elements, itshould be understood that those acts and those elements may be combinedin other ways to accomplish the same objectives. With regard toflowcharts, additional and fewer steps may be taken, and the steps asshown may be combined or further refined to achieve the methodsdescribed herein. Acts, elements and features discussed only inconnection with one embodiment are not intended to be excluded from asimilar role in other embodiments.

For means-plus-function limitations recited in the claims, the means arenot intended to be limited to the means disclosed herein for performingthe recited function, but are intended to cover in scope any means,known now or later developed, for performing the recited function.

As used herein, whether in the written description or the claims, theterms “comprising”, “including”, “carrying”, “having”, “containing”,“involving”, and the like are to be understood to be open-ended, i.e.,to mean including but not limited to. Only the transitional phrases“consisting of” and “consisting essentially of”, respectively, areclosed or semi-closed transitional phrases with respect to claims.

As used herein, “plurality” means two or more.

As used herein, “and/or” means that the listed items are alternatives,but the alternatives also include any combination of the listed items.

What is claimed is:
 1. A system for real-time participation within amedia presentation, comprising: a compositing processor programmed to a)generate a participant overlay by preprocessing said media presentationto extract a time-variant metadata stream associated with one or moreinteractive regions depicted in said media presentation, wherein saidparticipant overlay includes simulated physics, wherein said simulatedphysics include boundaries of said one or more interactive regionsdefined in accordance with one or more content elements in said mediapresentation; b) capture a participant's interaction with said mediapresentation through said one or more interactive regions of saidoverlay; and c) combine said overlay with said media presentation fordisplay to said participant on a display.
 2. The system of claim 1,wherein said participant's interaction comprises any of a user actionupon said one or more content elements in said media presentation, auser action expressed in said media presentation through an avatar, or auser action in said media presentation in connection with a game element3. The system of claim 1, further comprising a display forsimultaneously displaying said media presentation and said participantoverlay.
 4. The system of claim 3, further comprising a controllerconfigured to facilitate the participant's interaction within said mediapresentation in real time.
 5. The system of claim 4, said controllercomprising: any of a game controller, joystick, mouse, touchpad,ultrasonic motion detector, microphone, and video camera.
 6. The systemof claim 1, further comprising: a plurality of participant locations,each participant location supporting at least one participant; and anetwork for interconnecting each participant location; wherein saidcompositing processor is configured to simultaneously captureinteraction of each participant at each of said plurality of participantlocations and add the interaction of each participant to said overlay inreal time for display at a display at each of said participantlocations.
 7. The system of claim 1, further comprising a participantlocation comprising a facility that supports a plurality of participantsand a display configured to display said media presentation and saidparticipant overlay simultaneously to each of said plurality ofparticipants.
 8. The system of claim 7, further comprising means foreach participant in said plurality of participants to interact with saidmedia presentation in real time.
 9. The system of claim 1, wherein saidcompositing processor comprises: a local processor at said participantlocation.
 10. The system of claim 1, further comprising: a gameprocessor programmed to provide at least one game play element to saidmedia presentation via said participant overlay.
 11. The system of claim10, wherein said game processor and said compositing processor comprise:one or more local processor at said participant location.
 12. The systemof claim 1, further comprising: a media processor programmed to providesaid media presentation to said compositing processor.
 13. The system ofclaim 1, wherein said compositing processor is further programmed tosynchronize captured participant interaction with said mediapresentation.
 14. The system of claim 1, wherein said compositingprocessor is further programmed to map said media presentation on apredetermined basis, to add interactive elements to said mediapresentation in accordance with said map, and to track participantinteraction with said interactive elements based upon said map.
 15. Thesystem of claim 14, wherein said compositing processor is programmed touse said map to track participant movement of said interactive and tomaintain accurate registration between said participant overlay and saidmedia presentation.
 16. The system of claim 1, wherein said participantoverlay comprises: an avatar representation of said participant.
 17. Thesystem of claim 1, wherein said participant overlay comprises:interactions of said participant with any of said media presentation andinteractive elements within said participant overlay.
 18. The system ofclaim 1, wherein at least one element of said media presentation isextracted therefrom and re-rendered into an interactive feature.
 19. Amethod for real-time participation within a media presentation,comprising: a) generating a participant overlay by preprocessing saidmedia presentation to extract a time-variant metadata stream associatedwith one or more interactive regions depicted in said mediapresentation, wherein said participant overlay includes simulatedphysics, wherein said simulated physics include boundaries of said oneor more interactive regions defined in accordance with one or morecontent elements in said media presentation; b) capturing aparticipant's interaction with said media presentation through said oneor more interactive regions of said overlay; and c) combining saidoverlay with said media presentation for display to said participant ona display, said interaction comprising any of a user action upon saidone or more content elements in said media presentation, a user actionexpressed in said media presentation through an avatar, or a user actionin said media presentation in connection with a game element; and d)simultaneously displaying said media presentation and said overlay onthe display.
 20. A game system comprising: a display; and a processorcoupled to the display, wherein the processor is configured to combinepre-existing video content with an interactive overlay that permits userinteraction with one or more virtual objects overlaid onto a scene ofthe pre-existing video content, wherein said overlay includes simulatedphysics, wherein said simulated physics include boundaries defined inaccordance with one or more content elements in said video content.